Evaluating Text-type Suitability for Machine Translation a Case Study on an English-Danish MT System

This paper reports on an evaluation of how well a specific MT system would perform in translating new text-types including an assessment of in what ways the system itself could be extended to deal with new text-types. The overall evaluation and quality criterion is defined in terms of how much effort it takes to post-edit the text after having been translated by the MT system. A structured questionnaire rating different error types was given to the post-editors involved. The results were then "translated" into a number of mainly linguistic phenomena occurring in the input text causing these errors. In order to achieve consistency and reliability the analysis of the new text-types was automated as far as possible. A suite of programs was developed, each of which identifies a particular phenomenon and assigns scores for each occurrence. A reference text, known as being a good text, was first analysed using the procedure in order to provide a benchmark against which to assess the results from analysing the new text-types. After running the evaluation, a representative subset of the new text-types were then selected and translated by a slightly revised version of the MT system and assessed by the post-editors (using the same questionnaire).